[GuideLLM Refactor] Fix from-file #366
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sjmonson
requested changes
Sep 24, 2025
Comment on lines
55
to
142
| DataType = ( | ||
| Iterable[str] | ||
| | Iterable[dict[str, Any]] | ||
| | Dataset | ||
| | DatasetDict | ||
| | IterableDataset | ||
| | IterableDatasetDict | ||
| | str | ||
| | Path | ||
| ) | ||
|
|
||
| OutputFormatType = ( | ||
| tuple[str, ...] | ||
| | list[str] | ||
| | dict[str, str | dict[str, Any] | GenerativeBenchmarkerOutput] | ||
| | None | ||
| ) | ||
|
|
||
|
|
||
| async def initialize_backend( | ||
| backend: BackendType | Backend, | ||
| target: str, | ||
| model: str | None, | ||
| backend_kwargs: dict[str, Any] | None, | ||
| ) -> Backend: | ||
| backend = ( | ||
| Backend.create( | ||
| backend, target=target, model=model, **(backend_kwargs or {}) | ||
| ) | ||
| if not isinstance(backend, Backend) | ||
| else backend | ||
| ) | ||
| await backend.process_startup() | ||
| await backend.validate() | ||
| return backend | ||
|
|
||
|
|
||
| async def resolve_profile( | ||
| constraint_inputs: dict[str, int | float], | ||
| profile: Profile | str | None, | ||
| rate: list[float] | None, | ||
| random_seed: int, | ||
| constraints: dict[str, ConstraintInitializer | Any], | ||
| ): | ||
| for key, val in constraint_inputs.items(): | ||
| if val is not None: | ||
| constraints[key] = val | ||
| if not isinstance(profile, Profile): | ||
| if isinstance(profile, str): | ||
| profile = Profile.create( | ||
| rate_type=profile, | ||
| rate=rate, | ||
| random_seed=random_seed, | ||
| constraints={**constraints}, | ||
| ) | ||
| else: | ||
| raise ValueError(f"Expected string for profile; got {type(profile)}") | ||
|
|
||
| elif constraints: | ||
| raise ValueError( | ||
| "Constraints must be empty when providing a Profile instance. " | ||
| f"Provided constraints: {constraints} ; provided profile: {profile}" | ||
| ) | ||
| return profile | ||
|
|
||
| async def resolve_output_formats( | ||
| output_formats: OutputFormatType, | ||
| output_path: str | Path | None, | ||
| ) -> dict[str, GenerativeBenchmarkerOutput]: | ||
| output_formats = GenerativeBenchmarkerOutput.resolve( | ||
| output_formats=(output_formats or {}), output_path=output_path | ||
| ) | ||
| return output_formats | ||
|
|
||
| async def finalize_outputs( | ||
| report: GenerativeBenchmarksReport, | ||
| resolved_output_formats: dict[str, GenerativeBenchmarkerOutput] | ||
| ): | ||
| output_format_results = {} | ||
| for key, output in resolved_output_formats.items(): | ||
| output_result = await output.finalize(report) | ||
| output_format_results[key] = output_result | ||
| return output_format_results | ||
|
|
There was a problem hiding this comment.
Lets move all of this to a new file (initializers.py maybe); entrypoints.py is kind of our public interface for ABI usage so I think its best only have the front-facing functions in here. Also allows for some type reuse in scenarios.py
There was a problem hiding this comment.
To bring the conversation online: I thought so too, but it appears that Mark would prefer them to be in the same file. To clarify, I added comments.
sjmonson
approved these changes
Sep 24, 2025
Signed-off-by: Jared O'Connell <joconnel@redhat.com>
Signed-off-by: Jared O'Connell <joconnel@redhat.com>
Also remove unused import Signed-off-by: Jared O'Connell <joconnel@redhat.com>
jaredoconnell
force-pushed
the
features/refactor-fix-from-file
branch
from
b17f644 to
f926e5b
Compare
sjmonson
approved these changes
Sep 24, 2025
jaredoconnell
merged commit ab5466b
into
vllm-project:features/refactor/working
8 of 17 checks passed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR ports the new functionality from
benchmark runtobenchmark from-file, and does so in a way that reuses as much code as practical to have one source of truth.Details
Test Plan
Run a benchmark with an output of json or yaml, and use
from-fileto re-import it and export it. You can select any output type supported bybenchmark run.guidellm benchmark from-file ./result.json --output-formats consoleguidellm benchmark from-file ./result.yaml --output-formats yamlRelated Issues
Use of AI
## WRITTEN BY AI ##)